Skip to content

perf: stop computing an HLL estimate on every merge and size() call - #7629

Open
comphead wants to merge 1 commit into
apache:mainfrom
comphead:hll-estimated-size-7341
Open

comphead wants to merge 1 commit into
apache:mainfrom
comphead:hll-estimated-size-7341

Conversation

@comphead

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #7341. #7341 stays open for item 2 (a GroupsAccumulator) and for the datasketches bump once a release includes apache/datasketches-rust#317.

Rationale for this change

#7345 fixed the problem in the title of #7341. Item 3 is still real on main: merging a partial sketch that is already dense costs about 270 µs at lgConfigK 18, while the register-wise max it needs takes 5 µs. datasketches 0.6.0 rebuilds its register cache, one f64 division per register, after every dense merge.

apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers that rebuild until the next query. Comet would not benefit from it. SparkHllUnion::merge calls estimate() on the union and on the incoming sketch before every update, to keep a dense flag for the heap-size inference, and heap_size() calls estimate() on every accumulator size(). Each of those calls would force the deferred rebuild.

datasketches 0.6.0 exposes the size directly. HllSketch::estimated_size() and HllUnion::estimated_size() read the coupon array or register array length in O(1), and the result is exact. The estimate-based inference predates that API, since 0.3.0 kept the layout private.

What changes are included in this PR?

  • SparkHllSketch::heap_size() and SparkHllUnion::heap_size() return the crate's estimated_size().
  • The layout module, both dense flags and the preamble module are removed, so merge and size() no longer compute estimates.
  • heap_size_follows_the_crates_layout now asserts that heap_size() equals the allocation at every step, instead of "never below, at most one size step above".

How are these changes tested?

The existing heap-size tests cover it. heap_size_follows_the_crates_layout now requires an exact match with the allocation the crate serializes at every cardinality, through each SET resize and the promotion. high_lg_config_k_is_charged_for_what_it_holds, union_heap_size_follows_the_crates_layout and singleton_groups_at_high_lg_config_k_fit_a_small_pool are unchanged. Sketch results do not change: only the size accounting is touched, and the probe below gave identical output bytes in every build.

The numbers come from a throwaway example (not in this PR) that runs a Final AggregateExec over partial states at lgConfigK 18, with rows interleaved one partial per group per batch, the way a reducer reads map outputs. Release build on an Apple M-series laptop, hll_sketch_agg, ns per partial row, median of 5 runs over two interleaved rounds. hll_union_agg gives the same picture.

With datasketches 0.6.0, as shipped by this PR:

Case main This PR
Tiny: 200K groups x 8 partials, 0 to 8 values 211 169
SET: 2K groups x 100 partials, 50 values 1,453 1,450
Small partials into dense groups: 20 x 400, 200 values 3,431 3,445
Dense partials: 8 groups x 32, 30,000 values 359,000 361,000

With datasketches at apache/datasketches-rust@5f080f98e5 (#317, still version 0.6.0, through [patch.crates-io], not part of this PR):

Case main This PR
Dense partials, per row 351,000 ns 24,200 ns
One SparkHllUnion::merge of a dense sketch 266 µs 5.5 µs

A plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge costs about what the merge itself has to do. Differences under 5% are within run-to-run noise here.

@github-actions github-actions Bot added enhancement New feature or request performance area:aggregation Hash aggregates, aggregate expressions labels Nov 14, 2026
@comphead
comphead requested review from andygrove and viirya November 14, 2026 22:54
`SparkHllSketch` and `SparkHllUnion` inferred their heap size from
`estimate()`, and `SparkHllUnion::merge` called `estimate()` on both
sides before every update to keep a `dense` flag for that inference.
`datasketches` 0.6.0 exposes the size directly: `estimated_size()`
reads the coupon or register array length in O(1) and is exact. Use
it, and drop the `layout` inference, both `dense` flags and the
`preamble` module.

On 0.6.0 this makes Final-mode `hll_sketch_agg` and `hll_union_agg`
over many tiny groups about 20% faster (211 to 169 ns per partial row
at lgConfigK 18). It matters more once Comet moves to a datasketches
release with apache/datasketches-rust#317, which defers the register
cache rebuild after a dense merge until the next query. The
per-merge `estimate()` forced that rebuild on every merge. Built
against apache#317, merging an already dense sketch drops from 266 us to
5.5 us, close to the 5.2 us a plain register-wise max takes.

Part of apache#7341.
@comphead
comphead force-pushed the hll-estimated-size-7341 branch from ef10fc2 to ffa0694 Compare November 15, 2026 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:aggregation Hash aggregates, aggregate expressions enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant